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ABSTRACT 


The procedure and data of the Second Annual Award for Excellence in Teaching 
are discussed and studied. Basic statistics of balloting are presented and increased partici- 
pation is noted. The techniques of data reduction are described. Some special studies are 
made concerning (i) score data has distribution of mixed exponential type (11) the effect 
on the score of whether or not an instructor teaches during the quarter of the balioting 
(iii) concordance determination with last year’s ranking of instructors. Recommendations 


for modifying future procedural items are made. 
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I. INTRODUCTION 


The Superintendent of the U. S. Naval Postgraduate School, Rear Admiral 
Robert W. McNitt, USN, directed in May of 1969, that an award be established to 
recognize excellence in teaching. To comply with his directives a committee was 
formed from among the faculty. The objective of this committee was to select a 
recipient each year for the Outstanding Teaching Award to be given during the June 
graduation. 

This committee was to be composed of a Chairman, a secretary and five members. 
These seven people were to be chosen to provide a wide variety of backgrounds and 
curricula. Each year four persons would leave the committee and be replaced by four 
new members. Thus, new ideas and concepts would be engendered in the selection 
process while retaining a core of persons familiar with the existing system. 

The members of this year’s committee were Professors Kovach (Chairman), 
Hoisington (Vice-Chairman and Secretary), Musgrave, Nunn, Renard, Read and Schultz. 
The committee decided to retain from Tt year’s committee the same philosophical 
outlook concerning the nature of excellence in teaching along with how it should be 
reliably identified and evaluated. The interested reader will find this outlook in 
Robert William Geary’s Thesis, The First_ Annual Award for Excellence in Teaching, 

A Study of Procedure and Analysis of Data, June 1970. However, to set the ground- 
work for this paper certain guidelines for the selection process will be discussed. These 
guidelines are as follows: 


1. All faculty with the exception of the Department Chairmen and the members 
of the selection committee are eligible to receive the award. 


2. The voting population includes all current students, faculty, Curricular 
Officers and Assistant Curricular Officers. Also included this year is the 
group of Naval Officers which eraduated in 1969. 


3. Participation in the poll is entirely voluntary and no one is required to 
sign his ballot. 








4. Care and discretion are to be used to ensure that no ranking of candidates 
is published. 


5. For the ballot to be valid the voter must indicate himself to be reasonably 
well acquainted with at least five members of the candidate list. 


6. The voting population is cautioned not to vote on the basis of personality 
or popularity but on teaching ability and performance. 


In order to maintain total objectivity among the selection committee it was 
determined that an individual from outside the faculty would handle the ballots, 
collate the data and submit the results to the selection committee. Only after the 
recipient of the award had been selected, would the winner’s name be made known 


to the selection committee. 








Il. THE BALLOT 





A. FORMAT 

An instruction sheet was attached to the front of the ballot to explain the purpose 
of the award, give instructions on how to fill out the baliot, and explain the require- 
ments for validating the ballot. Results indicate that greater care should be exercised 
in preparing the cover sheet, particularly in the area of ballot validity. This is covered 
in greater length later in this section. 

The ballot itself was composed of four parts. Part A was a statistical gathering 
aid to help in an analysis of the voting population characteristics. The voter was 
asked to identify his voter category, rank, Branch of Service, and Curricular area if 
military; if civilian his academic rank and department were to be circled. 

Part B included three blanks to be used in indicating first, second, and third 
choices for the award using the four digit identification number in Part C. 

Part C of the ballot identified the candidates for the voter. Each eligible faculty 
member was listed in alphabetical order under the heading of his department. A four 
digit identification number preceded each candidate. The voter was asked to circle 
this identification number of all those faculty members with whom he considered 
himself to be reasonably well acquainted in an academic capacity. 

The last portion of the ballot, Part D, was to be used to comment on the reasons 
for his selection. This portion was not mandatory, but was considered beneficial in 
the final selection processes. 

The major variations in this year’s ballot from the year before were as follows: 


1. The second portion of the ballot (the blanks) was added to provide a 
faster and safer means of extracting the nominations from the ballot. 


2. The blanks were removed from the left side of the four-digit identification 
number. 








The only changes recommended for the actual ballot for next year would be to 
include another blank in Part B that can be marked with a “1” if a comment is made 
in Part D and to assign each teacher a permanent four digit identification number for 
Part C instead of renumbering each year. This would further facilitate data retrieval. 


Included as Annex A of this document is a FY ’71 ballot. 


B. THE BALLOTING PROCESS 

The method whereby the ballots were distributed and returned merits attention, 
for the selection process is only as good as the response of the voters. 

Last year the ballots were distributed through the Curricular Officers and Section 
Leaders. In an attempt to increase student participation the selection committee 
decided a new method should be found for distributing and returning the ballots. 

The ballots were distributed utilizing the Student Mail Center (SMC) this year. The 
completed ballot was placed in an envelope addressed to the Superintendent of the 
Naval Postgraduate School and deposited in a marked box outside thc SMC. The 
envelopes were coded 55 BAL to help route them to the person assembling the data. 

The Alumni selected to participate were the Naval Officers who graduated in 
calendar year “69. Their names and addresses mere taken from a data tape maintained 


in the computer facility after the addresses had been updated [ref. 4]. The completed 


ballots were returned to the mailing address previously mentioned. 


C. BALLOT VALIDATION 
1. Assembling the Data 
Once the completed ballots had been returned to a central location, the 
validation process for each ballot began. The ballots were removed from their envelopes 
and cover sheets were discarded. 
The validation process encompassed counting to ensure that Part C of the 
ballot had at least five identification numbers circled and that Part B did not have 


any write-in nominations. If less than five identification numbers were circied in 
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Part C or if the only nominations were write-ins, the ballot was considered void and 
placed in a separate stack. Valid ballots were numbered so that specific data could 
be retrieved at a later date if necessary. 

The chief cause of void ballots was the restriction that at least five persons 
be known by the voter. Of the 281 void ballots, 265 were deemed void for this 
reason. 

Another fifteen ballots or 5.7% of all void ballots were deemed void 
because write-in nominations of ineligible faculty had been made in place of numbers 
from the eligible faculty. 

One reason for the high number of void ballots, 281 out of 1722, could be 
attributed in part to the influx of first quarter students. Each person receiving a 
ballot was requested to return the ballot whether or not he knew five eligible faculty 
members. Yet, the number of void ballots is too great to be the result of first quarter 
students alone. Therefole, it Was concluded that 2 large number of persons misinter- 
preted the instructions. 

9 Transfering the Data to ‘Tape 

When the above validation process was completed, the ballots were given 
to key punch operators to transfer the data to IBM cards. The IBM cards were used 
as data for a COBOL routine to check for key punch errors and transfer virtually 
error free data to a magnetic tape. 

An explanation of how the COBOL routine checked for key punch errors 
will also clarify why the identification numbers were used. The four digits comprising 
the teacher identification number were arranged so that the first three digits identified 
the particular teacher. The first teacher on the ballot had an identification number 
101. One was added to this number to form the identification number for the second 
teacher, 102. The process was repeated until the last identification number was formed. 
The first three numbers formed a vector, N, which when multiplied by a fixed 


weighting vec‘or, 5, formed an inner product. The last digit of this inner product 


ili 








formed the fourth digit of the identification number. As an example teacher 299 would 
yield 
3 
(299) | SP] 6 45s 
it 
Thus, his full number would be 2990. 
The above described procedure proved very useful for only correct identifi- 


cation numbers were transferred from the ballot to the tape. An error in key punching 


would cause an error message giving the ballot number and its cause for rejection. 
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Ii]. STATISTICAL ANALYSES 


A. POPULATION CHARACTERISTICS 
One of the most important factors of the voting was the voter response. After 
sorting the valid from void ballots the percentages of response were determined for 


each voter cagegory. Table I below lists findings from the breakdown of the ballots. 


Table 1. BALLOT BREAKDOWN 


Vem Gatevow No. Eligible No. Valid % Valid % \ncrease Over 

Voters Ballots From Eligible Previous Year 
Student 1890 1023 54.1 Sal 
Alumni 25 246 46.9 12 
Faculty 314 Oy 34.1 8.5 
Curricular Officer 16 2 les (ley) 
Unknown* — 60 


* Includes 40 ballots in which Part A was omitted 


e% 


() indicates negative number 


The last column of Table I presents the percentage change over the last year, indicating 
that all valid voter categories, except that of the Curricular Officers!, improved. 
The number of eligible voters in both the student and faculty categories increased 
appreciably; therefore, the percentage increases shown indicate a significant increase 
in response. 

This year before mailing the alumni ballots, the address tape maintained by 
the computer center was updated. Even though five less ballots were mailed the 
increase in response was 7.9 percent. This supports the belief that updating of 
the tapes would improve alumni response. 

The large number of ballots with no voter category indicated this to be a 


definite problem area. Two possible causes are listed on the following page: 





1 Curricular Officers who identified themselves as alumni had their ballots included 
in the alumni group. 
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1. The instructions concerning the voter category were vague and misinter- 
preted. 


2. The voter did not want to take the time to fill out Part A. 

3 The voter did not want to be identified. 
Of the above the most likely explanation concerns the instruction sheet. The reference 
to Part A was given under Step O. This may have led the voter to assume that no 
importance was placed on this information. However, in order to improve the 
balloting procedure, information concerning the voters is necessary. The format of 
Part A should be changed to place the voter category under the heading of VOTER 
CATEGORY instead of beside the heading in hopes that the voters’ attention will be 
called to the choices. 

The 1441 valid ballots consisted of 71 percent students, 17.1 percent alumni, 
7.4 percent faculty, 0.3 percent Curricular Officers and 4.2 percent unknown. The 
420 ballot increase over last year reflects both an increase in voter population and 


improved participation in the voting. 


B. CLOSER OBSERVATIONS 

After studying the voter categories it was determined to research the military 
and faculty data for further information concerning the voter population. Table Il 
presents a breakdown of faculty voters by curricular department and academic rank. 
The columns indicate the different ranks and rows indicate academic department. 
By scanning the jth column and ith row, a determination of the number of valid 
ballots from faculty of i department and of j rank can be made. 

Tables III, IV and V breakdown the military voters by Branch of Service, Rank 
and Voter Category respectively in each curricular area. The ith row gives the 
curricular area and the jth column gives Branch of Service, Rank, or Voter Category 
according to the title of the table. The “faculty” column in Table V, Voter Category, 
contains those ballots marked faculty and military. One possible explanation for a 


person being listed as military and a member of the faculty would be that the voter 
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Table !11. FACULTY 
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Enrolled 
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Table 11!. BRANCH OF SERVICE 
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No Curriculum Note 


Operations 
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Operations 
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Computer Science 
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Table IV. RANK 
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Table V. VOTER CATEGORY 
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was one of the school’s Navy Instructor personnel. Another possible explanation 
would be an error in completing Part A of the ballot. However, the first explanation 
was deemed to be the more probable of the two. Figure 1, Recommended Part A, 


is a possible solution to some of the previously mentioned problems in this area. 


Voter Category 


S| Slfeletane 
F Faculty 
A Alumnus 
C Curricular Officer 
Students, Alumni, Curricular Officers Faculty 
Rank Branch of Service Curricular Area Rank Department 
O01 N_ Navy 30 Ops Analysis IR Instructor AE Aeronautics 
02 M _ Marine Corps 31. Aero Engr AT Asst Prof AO Aviation Safety 
03 A Army 32 Elec & Comm Engr AC Assoc Prof MN Bus Ad & Econ 
04 C Coast Guard 33 Ordnance PR Protessor Eb Ss elecsencn 
O05 -L_ At Force 34 Naval Engr GH Govt & 
06 F ~ Foreign Benin oct Humanities 
07 36 Mange & Comptr Sci MC Mat Sc & Chern 
3) ene oe! MA Math 
38 Baccalaureate ME Moch Engr 
39 Def Management MR Meteorology 


OC Oceanography 
OA Ops Analysis 


PH Physics 
NS Navy Met Sys 
Center 


Figure 1. RECOMMENDED PART A 


C. DISTRIBUTION OF M, 

An interesting statistic concerning the voting population was the number of 
eligible faculty members known by the ith voter. As the it voter becomes acquainted 
with more faculty, he should be able to choose ee for he is more familiar with 
the teaching methods of the eligible population. 

Figure 2 presents a histogram of M;, the number of eligible faculty members 
known by the ith voter. The mean or average number of faculty members known by 
the voting population was 16.8. At first slance this appears to be a good number, 


then there was the realization that a ballot must have hade at least five eligible ‘aculty 
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members marked known before it was considered to be valid. This fact along with a 
standard deviation of 9.7 showed that it was not safe to assume the average voter’s 
choice to be founded on a large base of experience. Until such time as a method 
can be devised to weight the ballots based on the voter’s experience, it should suffice 
to say that considering the wide range of voter experience (1 to 8 quarters) a large 


varience, 94.7, in M; was and can be expected. 


400 
E(M;] = 16.8 
TM, == 
300 
o 
= 
Oo 
> 
2 200 
He 
100 
| 
| 
| 
| 
0 js 
5 2s 50 ii 100 


Mj 


Figure 2. HISTOGRAM OF M; 


fee DISTRIBUTION OF Nj. 

Now that an indication of how many faculty were known on each ballot has been 
given, it would prove useful to discuss how many ballots indicated acquaintance with 
each faculty member. Figure 3 gives a histogram of Nj, the number of voters 
identifying teacher k. Table VI presents the frequency table used for this discussion. 
The mean number of voters who know teacher k was 82.9 with a standard deviation 
of 62.5. Of special interest was the fact that 514 people indicated they knew one | 


teacher. Due to this phenomena the teacher would have had to have significantly 
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more votes than the others in order to place high in the standings. This pointed out 
one of the major weaknesses of the present system of selection which is expounded 


upon in subsection E Determination of Sy. 


70 
60 
50 


40 


Frequency 


50 100 150 200 250 300 350 400 450 500 550 
N 
k 


Figure 3. HISTOGRAM OF Ny, 


Table VI. FREQUENCY TABLE OF Nx 


CELL FREQUENCY 

1- 24 22 
26- 50 64 
51- 75 56 
76-100 61 
101-125 44 
126-150 16 
ISS 11 
176-200 6 
201-225 5 
226-250 3 
251-275 0 
276-300 1 
301-325 0 
501-515 1 
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E. DETERMINATION OF S;, 

After the balloting data had been transfered to magnetic tape, a series of computer 
programs were run to determine specific statistics from the data. Appendix B contains 
a listing of these computer programs. One of the first important statistics determined 
was the score, S;, of each professor. In order to perform this exercise the aata was 
transferred to magnetic tape by the use of the computer routine “TRANSFER”. A 
second computer routine “PROFSORT” was run to rearrange the data from ballot 
format to a format identifying each teacher, T\,. That routine also gave a frequency 
distribution of M;. After the “PROFSORT” routine had been completed, the third 
routine, “SCORSORT” was run giving a printout of each teacher and his respective 
score which had been determined as follows: 


1. Teacher k’s first, second, and third place votes xj, X9 and xX respectively 
were divided by the number of persons identifying teacher k in Part C of 
the ballot. This yielded a vector V,. 

2. The vector V;, was multiplied by a weighting vector W and the product 
yielded S;, the score for teacher k. Mathematically this operation was 


expressed as 


The following table presents a summary of the “SCORSORT” printout showing 


k, Sp, Zia. Xa: Xone and X91 respectively. 


Table VII. SCORSORT SUMMARY 


k Sk Nk Zy Xk X9k X3 
1 DADV21 66 140 * 26 16 4 
2 2.08955 134 280 59 18 g 
3 1.84000 25 46 g 6 2 
4 1.82692 104 190 38 14 10 
5 1.77419 31 55 11 5 1 
6 1.69474 95 161 29 18 9 
7 1.68831 77 130 27 9 4 
g 1.63134 217 354 58 47 28 
9 1.59420 69 110 20 12 6 
10 1.55556 99 154 25 22 10 
11 1.53704 54 83 15 . 8 7 
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Table VII. SCORSORT SUMMARY (continued) 
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56 
57 
58 
59 
60 
61 
62 
63 
64 
65 
66 
67 
68 
69 
70 
71 
1D 
73 
74 
75 
16 
77 
78 
79 
80 
81 
82 
83 
84 
85 
86 
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38 
39 
90 
91 
92 
93 
94 
95 
96 
97 
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Table VII. SCORSORT SUMMARY (Continued) 


+k 


0.62821 
0.60938 
0.58511 
OSes 
0.57463 
Osta 
0.57143 
0.56522 
0.56436 
0.56383 
0.56250 
0.56000 
0.55670 
0.54369 
0.53226 
0.51961 
0.51948 
G5 lS 
O12 16 
O24 
0.50769 
0.50549 
0.50000 
0.48837 
0.48780 
0.47500 
0.47287 
0.47170 
0.46087 
0.46067 
0.45455 
0.44828 
0.43662 
0.42991 
0.42391 
0.42308 
0.41758 
0.41045 
0.40909 
0.40288 
0.39286 
O522386 
0.39024 
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64 
94 

109 

134 
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77 
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101 
94 
48 
50 
97 
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62 
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77 
33 
76 
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65 
91 
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43 
82 
40 
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53 
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44 
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92 
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Table Vil. SCORSORT SUMMARY (continued) 


ik >k Nk Zk Xk «= Xk Ak 
99 0.38710 124 48 6 8 8 
100 ~—-0.37975 79 30 3 6 6 
101 0.36364 77 28 5 2 4 
102 ~—- 0.36232 69 25 4 2 5 
103. 0.36184 [De ee SS 6 12 7 
104 0.35928 167 60 4 1 22 
105 0.35821 67 24 4 3 2 
106 —-0.35135 111 39 3 10 7 
107 ~—-:0.35099 151 53 8 i 7 
108  —-0.34286 140 48 6 9 6 
109 —- 0.33696 92 31 5 4 3 
110 ~—-0.33333 117 39 2 9 13 
114 0.33333 36 12 2 2 0 
Meee Oeaees 33 1 2 0 3 
ieee 08323 30 10 1 1 4 
114. 0.33000 100 33 3 6 9 
115. 0.32432 111 36 5 5 6 
Mio) Osteen 104 33 5 3 7 
117——-0.31707 41 13 2 1 3 
118 6.31690 142 45 6 8 5 
119 0.31646 79 25 2 6 5 
120 —-0.30909 110 34 5 6 2 
121 0.30769 65 20 0 7 6 
122 0.30081 123 37 7 3 3 
123. —-0,29412 68 20 1 5 6 
124 0.29412 17 5 0 2 
125 0.28571 35 10 2 1 0 
126 0.28523 298 85 1 15 1 
127 0.28205 39 11 2 1 1 
128 0.27922 154 43 7 6 3 
129 0.27692 65 18 3 2 2 
130. —-0.27500 40 1 2 0 3 
131 0.27027 185 so, 8 5 8 
132 0.25806 62 16 2 4 0 
133. 0.25641 39 10 1 2 2 
134. 0.25352 71 18 2 3 4 
135. 0.24390 41 10 1 2 2 
136 0.24324 37 9 2 1 
137 —-0,.24227 194 47 5 10 7 
138 0.23256 86 20 2 3 6 
139. 0.22951 61 14 1 2 6 
140. 0.22727 66 15 0 6 3 
141 0.22727 22 5 1 0 1 
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142 
143 
144 
145 
146 
147 
148 
149 
150 
131 
152 
153 
154 
ls 
156 
ey 
158 
132 
160 
161 
162 
163 
164 
165 
166 
167 
168 
169 
170 
171 
2 
les 
174 
1a 
176 
Wy 
178 
la 
180 
181 
182 
183 
184 


Table VII. 


Sk 


0.22642 
0.22500 
0.22222 
0.22000 
OrZze5 
0.21698 
0.21384 
0.21053 
0.20988 
0'20952 
0.20588 
0.20561 
0.20000 
0.19863 
0.19780 
0.19091 
0.18288 
0.18000 
0.17647 
O.17145 
0.17045 
0.16867 
O. TGS 
0.16667 
0.16667 
0.16250 
0.16250 
Onl si/ 
Vals 126 
0.14894 
0.14815 
0.14783 
0.14516 
0.14493 
0.14458 
0.13861 
0.13580 
0.13433 
Oals333 
0.12903 
0.12821 
O82) 
0.12745 


SCORSORT SUMMARY (continued) 
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53 
40 
iiez 
100 
64 
Ze 
153 
114 
§1 
210 
68 
107 
50 
146 
FA 
110 
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185 
186 
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190 
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194 
JS 
136 
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Table VII. SCORSORT SUMMARY (continued) 


Dk 
0.12727 
0.12658 
0.12381 
0.12245 
0.11905 
0.11864 
0.11538 
0.11483 
0.11215 
0.11009 
0.10784 
0.10588 
0.10526 
0.10448 
0.10345 
0.10345 
0.10300 
0.10227 
0.10145 
0.10112 
0.10000 
0.10000 
0.09756 
0.09655 
0.08738 
0.08333 
0.08333 
0.08333 
0.08333 
0.08163 
0.08000 
0.07843 
0.07813 
0.07692 
0.07563 
0.07547 
0.07500 
0.07407 
0.07143 
0.07092 
0.06993 
0.06944 
0.06849 


Nk 
55 
79 
105 
49 
42 
118 
52 
209 
107 
109 
102 
85 
76 
67 
58 
58 
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228 
220 
230 
231 
232 
255 
234 
235 
236 
237 
238 
280 
240 
241 
242 
243 
244 
245 
246 
247 
248 
249 
250 
Zo 
252 
253 
254 
229 
256 
257) 
258 
259 
260 
261 
262 
263 
264 
265 
266 
267 
268 
269 
2/0 


Table VII. SCORSORT SUMMARY (continued) 


> 
0.06667 
0.05941 
0.05195 
0.05000 
0.05000 
0.04977 
0.04918 
0.04444 
0.04054 
0.04000 
0.03896 
0.03846 
0.03704 
0.03636 
0.03265 
0.03175 
0.03125 
0.02632 
0.02542 
0.02506 
0.02353 
0.02174 
0.02105 
0.02000 
0.01563 
0.01493 
0.01136 
0.01111 
0.01053 
0.00909 
0.00000 
0.00000 
0.00000 
0.00000 
0.00000 
0.00000 
0.00000 
0.00000 
0.00000 
0.00000 
0.00000 
0.00000 
0.00000 
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15 
101 
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20 
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183 
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81 
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40 
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41 
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Table Vil. SCORSORT SUMMARY (concluded) 


kK Sk Nk Zt X1k Xk X3k 
Fn 0.00000 39 0 0 0 0 
5p 0.00000 35 0 0 0 0 
256 0.00000 34 0 0 0 0 
274 0.00000 32 0 0 0 0 
aS 0.00000 31 0 0 0 0 
6 0.00000 29 0 0 0 0 
Die 0.00000 29 0 0 0 0 
278 0.00000 28 0 0 0 0 
279 0.00000 27 0 0 0 0 
280 0.00000 23 0 0 0 0 
281 0.00000 23 0 0 0 0 
282 0.00000 20 0 0 0 0 
283 0.00000 19 0 0 0 0 
284 0.00000 18 0 0 0 0 
285 0.00000 14 0 0 0 0 
286 0.00000 14 0 0 0 0 
287 0.00000 9 0 0 0 0 
288 0.00000 6 0 0 0 0 
289 0.00000 6 0 0 0 0 
290 0.00000 2 0 0 0 0 


S;, is related to another statistic, Z;, defined by 


3 
7 xe 
k i=l ik 
It is easily recognized that 


aia 


Se Ny, « 


An analysis was conducted of the relationship between Z;. and N;,, where Z), was 
the total points received by teacher k and N; was the number of voters familiar with 


teacher k. A scatter diagram, Figure 4, was constructed and it was concluded from 


this diagram that no relationship more sophisticated than Le 5), Wy. was warranted. 


F. DOMINANCE 
The data from the score statistics was used to generate other statistics to be used 


for award recipient selection. A method was needed to reduce the number of elizible 
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Figure 4. SCATTER DIAGRAM OF 2, vs N, 


30 


160 








recipients without resorting to subjective guesses. Two methods for doing this were 
absolute dominance and complete dominance. These two concepts are discussed below. 
1. Background 
Associated with each teacher k, T},, are statistics which give specific information 
about the individual. These include: 
Xj, = the number of jth place votes received by T), 
N,. = Number of voters knowing T;, 


Wigan Xu /Ny, = fraction on ith place votes awarded to T;, from the 
population that knew him 


W =a weighting vector 


The following structure for a preferential region was suggested by Professor 
T. Gawain, Chairman of the first Selection Committee. It would seem logical that W 
would be constrained such that | 
W17 Wo > Wg ae 
since a first place vote should be worth more than a second place vote which in turn should 
be worth more than a third place vote. Further, since weighting all place votes the sam2 
or giving one of the places a zero weight would be contrary to requesting three separate 


rankings, there must be greater restrictions on W. In fact, it would seem logical to have 


where (a1, by) and (ao, bo) define a preferential region interior to the unit square. The 
S 
values of aj, a9, by and bo would be chosen based on a determination of the smallest 
and largest ratios acceptable for the three rankings. This study used 
ay = ag = 1/3 
and 


by = bo = 2/3 


as the acceptable parameters. 


ol 





2. Development 
It is desired to compare T},’s score, S},,with other teachers scores, S;,’. The 


main interest lies in all values of W such that 


Sk > Sy’ 
or 
wiVix t WoVon * W1V aK 2 1V1k + W2Von * W3V 3K (3) 
for all 
he 
rearranging (3) yields 
wa(Vaie Vin’) + WolVon-Von’) + W3(V3K-V 3k) 2 9- (4) 


Now if all values of V;, - Vi,’ > 0, i = 1,2,3, then Sj, > Sj,’ regardless of W and S,’ 
can be eliminated from competition. Thus, if T, has a vector V), such that vee Ve 
i= 1,2,3, then T), absolutely dominates T),’. This form of dominance is absolute 
dominance. 

The next form of dominance eliminates all teachers who, in order to win, 
must have weighting systems that violate wy > Wo > Wg. Divide (4) by wo and 
rearrange yielding 

wcieg Senet “2 Vox Yok if Vo, - Va,’ > 0, 


Wo : Vor-V3x W9 : Var-V3k otherwise the sense of the 
inequality is reversed 


(5) 


Since the V;,, and Via values are known, the dominance relationship is determined 
by the values of wj, i = 1,2,3. For the appropriate values Oly sales 1,2,3, Ty, is not 
dominated by T},’. 

Let y = W/W» and note that O0< y< 1. Let X = wy/Wo and note that 
1< X Then (5) becomes 

NneVir | Yaw Vex Conia 10: 
~~ Vana V3x-V3k 
Figure 5 gives a plot of y vs x. The shaded area indicates the feasible region 


for weights. 


32 








\\ 


Lidl 


1 2 3 


Figure 5. DOMINANCE PLOT 


Now consider the boundary 


| aie .y, SB ranor 
V3k V3k Vak-V3k 
This is the slope intercept form of the equation for a line. Analysing this indicates 


that for T;, to be completely dominated by Ty.’ either: 


Va1-V 

= 50 ancy) = 1, for Va, - Vay’ > 0 
V3k°V 3k 
Ven 

ye eee ane anes 0} for V3, - Va,’ <0 
V3k-V3k 


3) Vax - Vek = 0 and 


Vox Vox 


a) Vik = pane < O and ViVi 


b) Vik - Voie = 0 and Vox, < Vox’ 


If any of these obtain, then T;, is completely dominated by T;,’ otherwise there is 
no decision. Tj, will be eliminated if he is completely dominated by at least one T),’. 
3. Results 
A computer routine was prepared based on the above theory and dominance 
tests were run. Table VIII presents a summary of the data used and the results of the 


dominance tests. The AD column lists the results of the Abolute Dominance tests 








employing a 1 to indicate T}. is absolutely dominated and a 0 to indicate it is not 


absolutely dominated. The CD column lists the results of the Complete Dominance 


tests using 1 and O as above to indicate complete domination. 
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Table VIII. 


Vik 


0.394 
0.440 
0.320 
0.365 
CDS 
0.305 
0.351 
0.267 
0.290 
e253 
0.278 
0:273 
0.290 
C.224 
V5226 
0.214 
0.240 
0.222 
0.189 
0.216 
0.182 
0.214 
0.200 
Galo 
0.160 
0.198 
0.146 
0.179 
0.154 
0.174 
OS 
0.146 
0.125 
0.154 
0.172 
Oul2 
0.098 


DOMINANCE TESTS RESULTS 


Vok 


0,242 
0.134 
0.240 
0.135 
0.161 
Oi 39 
0.117 
O27 
0.174 
eZ 2 
0.148 
0.114 
0.118 
0.204 
0.204 
0.143 
0.116 
0.140 
0.189 
O36 
0.089 
0.107 
0.096 
0.098 
0.176 
0.063 
0.146 
0.090 
0.154 
0.109 
0.070 
0.115 
0.167 
0.088 
0.069 
0.142 
O59 
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V3k 


0.061 
0.060 
0.080 
0.096 
0.032 
0.095 
0.052 
O29 
0.087 
0.101 
Gale 
0.182 
0.075 
Oe 
0.108 
0.143 
0.085 
0.104 
Oe 
0.043 
0.174 
0.018 
0.096 
0.098 
0.080 
0.104 
0.1 ley 
0.090 
0.058 
0.043 
0.105 
0.104 
0.083 
0.099 
0.034 
O.007 
0.098 
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Table VII]. DOMINANCE TESTS RESULTS (concluded) 


Ty Vik V9oK V3E AD CD 
38 0.083 0:179 0.107 1 1 
2 0.071 0.143 0.214 0 1 
40 0.158 0.070 0.000 | 1 
4] OF 0) 0.103 0.083 | 1 
42 0.143 0:05), 0012 | 1 
43 0.120 0.090 0.040 | 1 
44 0.000 0.250 0.200 0 1 
45 0.044 Oa222 0.067 1 1 
46 0.089 0.089 0.156 1 1 
47 Onizs 0.063 0.063 | 1 
48 0.109 0.082 0.082 1 1 
49 0.041 0.189 Om35 0 1 
50 0.126 0.058 0.049 | 1 


G. PAIRED COMPARISONS 
1. Development 
Assume a comparison of two teachers, Ti, Ti,’, under the constraint that the 


voters must know each of them. Let 


No 


Nie = 2. Yik Vik’ 
el 
where N,, = total number of ballots and yj, 1s an indicator function whose value 
is one if ballot i was acquainted with T). and zero otherwise. Nk? represents the 
number of voters who knew both T, and T,». Note that Nik? is a symmetric matrix 
with zeroed diagonals. 
In order to determine the scores of T). and Ty», first determine Z.(k,k’) 
and Z;,(k’,k) where Z),(k,k’) represents the total score accumulated by T;, from the 
population of votes that knew both T). and T,.». Thus, 


N 
O x 
Z,(K,k’) = 2 Yaya (OT aik ee le | sa) 
1 hee 


and 


N 
Zy(k’,k) = y VinVik? (Wi XGK + WoXjoK' + W3X43k) 
]= 
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where, Aik is the number of ju place votes received by T;, on the ith ballot. 


Then, 

S},(k,k’) = Zy,.(K,k’)/Ny > 
and 

Sy(IK) = Zi KY/NY 


S;,(k,k’) represents the score of T), under the constraint that the voters 
knew both T, and T,,». A similar interpretation can be made for 5), (k’,k) and Ty’. 
Now a fraction 5}.(k,k’)/S;.(k’,k), is formed so that 

1) If S,.(k,k’) > S)(k’,k) the value is greater than one. 

2) If S,(k,k’) << S,(k’,k) the value is less than one. 


3) If S,.(k,k’) = S,(k’,k) the value is one. 


A reversal is noted if S; > S,’ and 2) above occurs and a tie if 3) above occurs. By 

reversal is meant, the paired comparison fails to confirm the primary ordering of the 

two teachers. The significance of a reversal is enhanced the greater the value of Nye 
A computer routine based on the above theory was run using the data and 

a comparison was made for ae k=1,..., 7. Table IX presents the results of that 


run. In the table N is the number of voters who knew both iy and Es: 


H. COMMENTS 

All individuals passing the dominance tests, zeros in both AD and CD, had the 
comment section of the ballots identifying them scrutinized and an abstract was made 
of the comments. In order to maintain complete objectivity of the selection committee 


all references to name or department were removed from the comment abstract. 


I. THE SELECTION 
Once all of the forementioned tests were performed and the results obtained, a 
set of statistics was prepared for the selection committee. This set of statistics 


included: 
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129/142 
140/160 
190/160 
Boe Oi 
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191/203 
133/033 
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17/067 
120/160 
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Table IX. 
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PAIRED COMPARISONS 


ip 
S/S, 


250/100 
250/045 
114/124 
082/106 
160/140 
227/109 
080/060 
111/111 
100/022 
088/013 
067/017 


V5 
$5/5 


200/144 
000/200 
000/067 
200/067 
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S/S, 


078/111 
221/079 
194/041 
193/027 
067/100 
080/080 


000/133 < 
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N Ty 
4 8 
4 10 
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N Ty 
20 17 
10 2 
10 8 
5 4 
5 9 
5 19 
4 13 
4 20 
3 19 
3 18 
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025/250 
025/200 
0337033 
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S6/S, 


095/050 


WAA 


ee 


060/080 < 


690/160 
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200/040 
080/000 
050/000 
125/000 
000/000 
033/000 
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1) 
2) 
3) 


4) 


The results from the score calculations 
The results of the Dominance Tests 
The results of the paired comparison test 


The comment abstract. 


After deliberating for some time on the data, a decision was made for To. That 


individual was identified to the selection committee as Professor E. C. Haderlie of the 


Department of Oceanography. 
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IV. FURTHER ANALYSES 


A. BACKGROUND AND SUMMARY 
It was observed last year by Geary [ref. 3] that the distribution of scores could 

be represented by the mixed density 

fo(s) = q for s = 0 

= ne : for cs) 0 
A 

where p+ q=l and A. > 0. The parameter values were a = 177 and A = 2.09 for 
a population of N = 249 scores. The following questions were raised: 


1. Does the above distribution type hold again this year? If so, have the 
parameters remained the same? 


9. Is the distribution of scores the same for teachers who taught the quarter 
of the balloting and those who did not? If not, do the teachers who taught 
score better? 


3 Are the teacher rankings produced this year in concordance Wits wlose 
produced last year? 


These questions are addressed in subsections B, C, and D. 
In summary, 


A 
1. The mixed exponential distribution still holds qith q = 114, A = 2.41 
and N = 290. The direction of the parameter changes was noteworthy in 
that it supported the hypothesis that the award was encouraging better 
teaching on a broad scale. The proportion of zero scores had decreased 
and the positive scores had smalier variability. 


2. The segregating of teachers into two groups, those who taught in the 
quarter of the balloting and those who did not, resulted in score distri- 
butions which were still of the exponential type, but the parameterization 
changed. It was believed that those who did not teach would have lower 
scores, but the results were not conclusive. This is discussed in subsection C. 


3 The Kendall Concordance Coefficient was computed. The value of OF it 


was mildly high, significantly above zero, but also significantly less than 
one. 
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B. SCORE DISTRIBUTION STUDY 
Figure 6 is a histogram of S;, the score of teacher kis = 0, for FY 712eNeite 
the reverse J-shape which indicates there is a good chance for the distribution of 
the scores to be exponential. The data for FY ’70 had a similar histogram and 
Lieutenent Robert Geary [ref. 3] showed that the data was exponential with a 
density function 
Fa(s) Oger een: 

A similar assumption was made this year and the maximum likelihhod estimation 
procedure for the parameter » was used. This results in 
where x is the mean of the sample of positive scores and is the parameter used in 
the exponential density function. Recall from Barr (ref. 1] that the probability 
density function of an exponential function is 

fy (x) = Ax. x >0 


The log of the likelihood equation is 


n fy - A2ZU;f 


where Uj is the cell midpoint and fi is the cell frequency. Setting the derivative of 


the above with respect to rN equal to zero yields 


“12 


j=l 


5 


thus 
A n 
Z 2 
A=1 Vad, U 


Tables X through XVII show the data breakdown of the FY 70 and FY 71 data. 


an 
Table XVIII shows the Avalues. 
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Table X. CH! SQUARE TEST DATA FY 70 ALL 






kK U; FOF, P; NP; F;/NP; 
1 1 05 42 2A 1894 38.83 45.43 
2 Z le 34 5.10 1601 32°52 S5ee2 
2 2 25 23 12 alee 25e20 20.94 
4 4 oD 2/ 9.45 e207 20.48 35.60 
5 5 45 13 De 0844 ie) og 
6 6 aoe) 13 alee 0650 1303 12.68 
1 1 65 5 3525) 0526 10.78 Zo 
8 8 Le 11 8.25 0427 els [exes 
9 9 ce Z 75 0361 7.40 6.62 
10 10 95 i 6.65 0278 5.10 8.60 
12 11 1.15 3 3.45 0415 Sol 1.88 
13 12 E25 5 O25 .0146 290 oe 
14 eS | los 
Is: 1.45 1 1.45 
16 le5> 1 > 
17 13 1.65 2 3.30 0359 i 3.40 
18 14 Ue 4 7.00 0051 1.05 15.24 
1S 1.25 0 0,00 | 
20 nee 0 0.00 
21 2.05 1 2.05 
ZZ Z\5 0 0.00 
23 22> 1 225 
24 Zoe) 2 7.05 0217 4.45 5262 









N=205 DU;f=96.25 
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Table XI. CH! SQUARE TEST DATA FY 70, 03, TEACHING 


j k U; F; UF; P; NP; Fi/NP; 
| 1 .05 31 155 .1894 2596 33.16 
3 3 25 18 4.50 1297 19.84 1Ge55 
4 4 3 22 148 .0999 15.28 31.68 
5 5 45 10 4.50 0809 12.38 8.08 
6 6 a 5 205 .0657 10.05 2.46 
7 7 {65 2.60 552 8.14 leo 
8 8 MS. SoZ .0449 6.87 ae 
9 2 85 Se .0347 Bye. | 9.23 


>) 


10 1 2p 6.65 .0280 4.28 11.45 








=“Ooo doe — - 0 Fyn = HIN mn F 
© 
© 
© 


14 1.35 

15 1.45 

16 1.55 1.55 

17 1.65 1.65 

we 120 as 7.00 0414 6.33 7.74 
19 1.85 0.60 | 
20 1.95 0.00 

21 2.05 0.00 

22 oe 0.00 

23 295 2.25 





_ 124 
XY =e» Ut. — Anise 
7 a 
j=1 
A hs 
As L= 212 
n 
2 13 9 
X73.2 = 3 FR INP, - N = 11.15 
k=1 
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Table X11. CHI SQUARE TEST DATA FY 70, 03, NOT TEACHING 


k UF Fo OU, P, NP; F /NP; 


2 oT 10.27 lees 


135 lEaZe Le 10 
NZ35 6.68 3./4 





















1824 
.0671 





3.49 18.34 





.Q952 4.95 





N=52 DU f=24.1 


_ 4 24 

Xx== Ui, = 46346 
n j=1 J 

A 

\ oe 016 
xX 


: - > F N = 7.82 
X7.2,.05 = 2, FL/NPL-N= 7. 
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Table XII. CH! SQUARE TEST DATA FY 70, O4, TEACHING 













k U; FOF, P; NP; F./NP; 
1 055... 30 150m Tie 28.44 31.65 
2 ise. 26°" 3:90) Bilbss 22.82 29.62 
3 nee SO 19.05 8.87 
4 B50 2 eo ene 14.54 30.33 
5 45 9 4.05 _.0811 11.68 6.93 
6 35 ie eo bear 9.37 12.91 
7 65 53.25 __.0523 fee 3.32 
8 75 5 3.75 __—0436 6.28 3.98 
9 85 a Ss ees 4.80 5.21 
10 95 5 4.75 __.0267 3.84 6.51 
1.05 os 
1.15 2 0 280 
[lene 3.3.75 __—.0530 7.63 4.72 
1,35 0 0.00 
1.45 ies 
1.55 0 0.00 
1.65 lees 
net Oe 2.98 | 
0 
0 
1 
0 
1 
2 


A 

d= L = 2.22 
xX 

; 13 


2 
X13-2,.05 = 2 FJNP, - N = 9.10 
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Table XIV. CHI SQUARE TEST DATA FY 70, 04, NOT TEACHING 


1 1 05 i .60 30 1055 13.65 
2 2 ip 8 leZ0 1495 9.14 7.00 
3 3 a 10 2) AZ ls liseo2 
4 + 2) 6 2.10 OS Onli? 5.82 
6 > 3D 2 1.10 1449 8.84 4.07 
8 6 i e 6 A a0n. sous 6.18 Drow 

85 2 ee) 

5 2 a0 

105 0 0.00 
zi el 1 lel 1149 feu 20 

les 2 ZU 

35 1 as 

145 0 0.00 
8 L355 1 lis 10525 eels 5.02 

1265 1 65 

1s 2 0 | 

1235 0 0.00 

oS 0 0.00 

2.05 0 0.00 

2.15 0 0.00 

aes 0 0.00 

1 Zoe 





N=61 2,U,F.=31.45 


7 24 
= UE. = Sis 
rT 


Solo 


= 1.94 


ae 
II 
|| 


9 
2 
X9.2.05 = 2 FLINPL - N = 3.75 
k=1 
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Table XV. CH! SQUARE TEST DATA FY 71, ALL 















k U; ° UF; Pi NP; F;/NP; 
1 1 05 53 2.65 2212 56.85 49.41 
2 2 15 51 7.65 1600 41.12 63.25 
3 3 25 31 7.75 1320 33.92 28.33 
4 4 35 Dy 9.45 1039 26.70 27.70 
5 5 45 jig Sale 0847 Die 14.88 
6 6 55 20 11.00 0636 16.35 24.46 
7 7 65 15 9.75 0501 ee 17.47 
8 8 75 5 3.75 0393 10.10 2.48 
9 9 85 4 3.40 0310 7.97 2.01 
10 10 95 7 6.65 0244 6.27 7.81 
11 11 1.05 6 ~— «6.30 0191 4.91 7.33 
1 
4 
0 
4 
16 1.55 3 4.65 
17 14 1.65 3 4.95 0104 2.67 13.48 
18 es 1 eS | 
19 1.85 p 2.70 
20 1.95 0 0.00 
1 
1 
N=257 LU;F/=106.65 
rr 
i aes 
n j=1 
A 
Az L = 2.41 
x 


2 eee 
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Table XVI. CHI SQUARE TEST DATA FY 71, Q3, TEACHING 





: kK U; Fo OU, P; NP; F INP; 
1 1 05 40 2.00  .2055 38.22 41.86 
2 2 aS 299. 435 1569 29.18 28.82 
A 4 35 22. #7.70 #441041 16 25,00 
5 5 ‘45 {Sunn Gust Uaepa 15.38 14.63 
6 6 55 14. 7.70 -.0631 11.74 16.70 
7 7 6S 138.45 0528 9.82 (Woe 
8 8 Es a oon eas 7.79 3.21 
0 10 95 54.75 0267 4.97 5.93 
ed ees 4 420  _.0212 3.94 4.06 
12 1.15 1) tats 
e115 33.75 0298 5.54 2.89 
14 135 0 0.00 
15 1.45 3 435 
16 155 ee ass 
ois tees ee ede 08 5.80 6.21 
18 eG 1) i 
19 1.85 2 3.70 
20 1.95 0 0.00 
21 2.05 1 2.05 
22 2.15 ee 2a 0211 
N=186 2.U;F=81.8 
ae 
xa l UF, = .439 
i=] 
No 
A= bk = 2.27 
xX 
; 14 


= z = 
X 14-2,.05 = 2 FL/NP, - N = 5.09 
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Table XVII. CHI SQUARE TEST DATA FY 71, Q3, NOT TEACHING 


05 VS oD 2442 17.34 eS 


ZZ 350 .1788 1226) 38.14 
11 .1410 10.01 12,09 






1844 13:09 4.89 





DOOCOAN KH DIR OTONNOAN AWM 


8.08 


ee 
x=—), UF; = 364 
ee 
j=1 
A 
ees 
X 


‘ai; 
Kaos = x, FL/NP, - N = 13.41 
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Table XVIII. VALUES 


eretp FY 70 faved. | 
Q3 Q4 Q3 
All Teachers 2.13 2a 2.41 
Those Teaching Zale 22 2007 
Those Not Teaching 2.16 1.94 o> 


A Chi-Square Goodness-of-Fit Test was used to test the hypothesis that the 
density functions determined using the parameters above were valid representations of 
their respective data. The null and aH ANE hypotheses for the above functions were 

Ho: fg(s) = eS 520 
Hy: Ho false, 
A 
where d is the value for the parameter of the density function, n is the number of 
positive scores, Ij, Io,... I, represents the scoring intervals, Pj = P(SEI}) under Ho, 
np; is the expected number of entries in I, under Hog and By is the number of observed 


J 


entries in [.. 


J 


Define 
r (F. - NP.) r re 
Z= y ac a = Y aa = IN 
j=l j j=l J 


where N = 2, Fi. As n gets large, z approaches the Chi-square distribution function 
J 
with r-2 degrees of freedom 
Thus let 
2 
P[X,.2 (1-q) 2 Obs Ts|Ho] =p 
Compare p withCland if 
fie py el then reject Ho, 
2) p = CL then accept Ho. 


The values determined for p were: 


00 





Table XIX. p VALUES 


Grote FY 70 EY al 
Q3 Q4 Q3 
All Teachers .084 .084 350 
Those Teaching 403 .613 oo 
Those Not Teaching .168 .800 .030 


Thus, it was concluded from the p values that Hg could not be rejected for 


reasonable levels of significance. 


C. TEACHING EFFECT 

The author felt a reasonable question arose during the data manipulation con- 
cerning the effect of teaching during the voting quarter on a teacher’s ranking. It 
seemed logical that a voter would be more likely to rank a teacher who was currently 
instructing than one who was off for the quarter. This assumption was based on the 
conjecture that a person is more likely to hold a stronger opinion of a teacher he 
is associating with at the time than one he has had in the past. 

To test this hypothesis it would be best to remove all non-student ballots, but 
this was not possible due to time constraints. The faculty from the Navy Management 
Systems Center were removed from the data for they do not teach by quarters. 

Care had to be exercised for during the FY 70 period the balloting was done at 
the start of Quarter 4, Q4. Since it was felt that strong opinions could not be 
formed during the first two weeks of a quarter, the data for Q3 was also gathered. 
The FY 71 period did not present the problem for the balloting was done near the 
end of Q3 for that year. 

Using the model 
fo(s)=q, s= 0 
= res s>0 
it is desired to test 


Ho: Oy = q9 and Ay == do 


ol 





against 

Hy: q9 > Gy and Ss Pe eI 
The author knows of no standard practice for this statistical problem. The following 
ad hoc discussion will shed some light on the question. 


A 
Reject Ho if 4p is sufficiently larger than a1 and NS is sufficiently 


larger that \ ie 
However, these estimates are not statistically independent and joint probability 
A A 
statements concerning them are difficult to calculate. In lieu of this, the pairs Nag 
A A 
are examined separately from qj, 49. 
A A A A 
Table XVIII showed that ne 2 4 in Quarter 3 of both years, but ae < oe in 
Q4 of FY 70. Thus, the direction of the shift was consonant with the discussion at 
the beginning of this section. To determine the significance of the change, some 
distribution theory is required. 


Since 


—= ———ew 
— 


A 
Ag _ ng Xy 
a Ny Ao 


where x; is the sum of positive scores for group i and n, is the number of positive scores; 


and since a positive score has an exponential distribution, it follows that 
Z 
Macow Pm; A), ox, ~ *(2n;) 


and it follows that A 


Ag 
F ‘ 
a a, 2n4,2Nno 
The p values are listed in Table XX. 


AN A 
Table XX. p VALUES FOR Ad/A, 


Year Quarter Test Statistic aul: Approx. p Value 
1970 3 Pale 2 = 1.019 104/306 40 
1970 4 1.94/2.22 = 0.874 122/288 5 
1971 5 2.75/2.27 = 1.211 142/372 105 


oO2 





Turning to qy and do, the normal approximation will be used in the two sample 


case. Let. 
h, = percentage of zeroes of those professors teaching during the observed 
quarter. 
ho = percentage of zeroes of those professors not teaching during the observed 
quarter. 
Then 
. nzhy Zz ngho 
h —— 
ny a No 


where h is the pooled percentage of all teachers, nj is the number of professors teaching 
during the observed quarter, and ng is the number of professors not teaching during the 


observed quarter. Now let 


G = Vb(1-h)(i/ny + 1/ng), 


then from Brownlee [ref. 2] 


(hg - 1/2ng) - (hy + 1/2nq) 
ge aa 
Ee 


70,1) 


using this formula, solving for p values yields 


Table XXI. p VALUES FOR q TEST 


Observed Quarter p Value 
Oa gn © 14 
Q4, FY 70 ag 
O32 Fea 09 


.< 
Again, the direction of the change supported the discussions at the beginning of 


this section, but the results were not significant using this as hoc approach. A sharper 
analysis may or may not give more significant support to this phenomenon. Such an 
analysis would include 

1. Removal of the non-student ballots 


hoe 
2. Correct joint probability statements concerning NX E> ant and do. 
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D. CONCORDANCE ESTIMATION 

Suppose that X = (X4, x5) has a bivariate cumulative distribution function 
F(x2,x)., then the vectors xj and X9 are said to be concordant if x4 - Ko and 
x4 - 5 have the same sign. Further, using Kendall’s [ref. 5] Coefficient of Concordance 
the commonality of two sets of data can be determined. If the test is performed and 
the coefficient of concordance is near one, the two sets of data show similar ranking. 
The further the coefficient is from one, near zero, the worse the relationship. Thus 
this test can be used to determine if a teacher scoring high one year is likely to score 
high the next. Also, it can be determined if the students rank the teachers the same 
each year. 


Kendall, [ref. 5], defines the relationship as 


128 


where S is the sum of squares of actual deviations, n is the number of rankings, m 
is the number of raters, T; = Dr tie - t), and t indicate ties. 

Table XXII is a 2 x 222 matrix of rankings determined from the scores.. The 
first row represents the ranking of teachers in FY 70 and the second row represents 
the ranking of the teachers in FY 71. The choice of 222 teachers was made so that 
the population would include only professors eligible both years. 


From this table the values of the equation take the form 


W = —y——12(2743884.46) _ 
(2)* [((222)°? - 222] - 12(2) (47838) 


yielding 
W = .77264 


Kendall also expressed a method to determine the significance of the Concordance 
Coefficient. Let 


2 
x2 = m(n-ty w= —228_ 
mn(n+1) 


where r = n-l, yielding 
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Total 
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Total 
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Table XXII. 
3 4 

5 12 

8 16 
14 15 
25 4] 
39 56 
25 26 
13 a2 
38 60 
36 37 
29 21 
65 58 
47 48 
111 83 
158 131 
58 59 
59 39 
ley 98 
68 70 
57 By 
125 107 
80 81 
123 135 
203 216 
91 92 
89 60 
180 oe. 
102 103 
82 116 
184 219 
113 114 
94 71 
207 os 
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204 145 
327 268 
134 135 
45 188 
179 323 
145 146 
159 102 
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2 
x, = 341.507 


2 
For values of r greater then 30, the quantity 2x, may be taken to be 


‘ 


distributed normally about the mean, vV2r-l1 . Thus 


2 
J 2x ~ N(V 2r-l, 1) 


or 

2 
or ; 9 

*921 - 21 N(0,1) 
So 


P(,/ 2x35, - 21 > 2) = .05 


or } foe, 
2X59, - 21> 1.65 


This specifies the critical region. Since 
341.507 > : 
. X99] = 256.51 
it was concluded that the resulting W = .77 is substantially significant. This 
indicated that the rankings of teachers was not random and that a teacher’s 
ranking was indicative, at least to a small extent, of the way the voting population 


as a whole felt about him. 
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V. CONCLUSIONS 


The ballot format changes discussed in Section I are recommended. It is also 
recommended that a year to year update of accumulated balloting statistics be ae 
as part of the automatic data processing. These include specifically 

1. Tabulation of voter characteristics. 

2. Computation of the mixed exponential parameters. 

3. Concordance estimates be determined. 

The study of the effect on the scores of teaching in the quarter of the balloting 
was inconclusive for the reasons given in Section IV. It is reeommended that plans 


be made so that this question can be firmly resolved in the future. 


08 





APPENDIX A 
SAMPLE BALLOT 


SUPERINTENDENT'S AWARD FOR EXCELLENCE IN TEACHING 


In order to augment the existing incentives for teaching of high quality, 
the Superintendent has established an award called the Superintendent's Award for 
Excellence in Teaching. The judgments of those who have participated in the educa- 
tional processes at the Naval Postgraduate School are of primary importance. This 
poll is being corducted to assess the collective opinion of students, faculty, staff, 
and selected alumni. After accumulating and processing the voting results, the 
Selection Committee will recommend an Awardee to the Superintendent. 


In view of the many measures of teaching excellence, there is no intent 
to fix the rationale by which each voter determines his nominations. However, in 
order to assist those voters who desire something more definite as a guide to their 
thinking, the following summary of some aspects of teaching excellence is offered as 
worthy of consideration: ‘The phrase, ‘excellence in teaching,’ refers to that com- 
plex of personal and professional qualities and actions on the part of the teacher 
which (a) make themselves felt primarily at the interface of personal contact between 
Student and teacher; (b) help transmute the student's encounters with his subject 
matter into insight, enlightenment and love of learning; (c) elicit from the student 
responses in thought, feeling and action which enhance his capacity for self education 
and (d) manifest themselves in an effective individual style which authentically re- 
flects the teacher's own unique personality, experience, character, and convictions." 


A meaningful selection of the Award recipient 1s heavily dependent upon 
the receipt of nominations from a large portion of the eligible "voters". Your 
response is therefore earnestly requested. Please complete the enclosed ballot in 
accordance with the attached instructions. The results of the polling, other than 
the Award recipient, will be treated as privileged information. 


BALLOTING PROCEDURE: 


Step 0. Complete the information requested at the top (Part A) of the ballot. 
These items are for purposes of statistical analyses only. 


Step 1. On the list of eligible faculty (Part C), encircle the four-digit 
identification number at the left of ail those with whom you are 
sufficiently acquainted to make a judgment. Use your own guidelines. 
For purposes of establishing a valid population base, it is important 
to encircle all with whom you are acquainted. It 1s the judgment of 
the Selection Committee that you should be acquainted with at least 
five eligible faculty members in order to cast a meaningful ballot. 
Accordingly, if you have circled five or more numbers on the ballot, 
proceed to Step 2. Otherwise, please return your ballot as directed 
in Step 4. 


step 2. From the subset of faculty you have indicated in Step 1, select from 
one (1) to three (3) nominees. Indicate your preferences in order 
by Placing the four-digit identification numbers of your nominees 
for first, second and third choices in the appropriate spaces pro- 
vided in Part B. : 


Seep 3. You are invited to furnish a short statement in Support of your 


primary nomination. Space is provided for this at the end (Part D) 
of the ballot. 


Step 4. Place your completed ballot in the enclosed return envelope, sign 


the envelope and return within ten days. Ballots will be Separated 
from the envelopes. 


Be 








DALLUI 


Part A - Statistical (encircle code at left) 


Part B - 


Part C - 


1014 
1025 
1036 
1047 


Department of Aeronautics 


1058 
R059 
1070 
1081 
1092 
1108 
9 
1120 
31 
1142 
1153 
1164 
75 
1186 
97 
1203 


Military 
Branch of Service 


Navy 

Marine Corps 
Army 

Coast Guard 
Air Force 
Foreign 


NI OPYR2 


First choice 


Voter Category 


OrpPnm 


Curricular Area 


Ops Analysis 

Aero Eng 

Elec & Comm Eng 
Ordnance 

Naval Eng 

Env Sci 

Mange & Comptr Sci 
Eng Sci 
Baccalaureate 

Def Management 


IR 
AT 
AC 
PR 


Second choice 


Student 
Faculty 
Alumnus 
Curricular Officer 
Civilian 
Rank 
Instructor AE 
Assist Prof AO 
Assoc Prof MN 
Professor EE 
GH 
MC 
MA 
ME 
MR 
OC 
OA 
PH 
NS 


Nominations (use the four-digit identification number) 
Third choice 


except department chairmen and members of the selection committee) 


Aviation Safety Programs 


Bomberger, R. B. 
Bradbury, C. M. 
mretcher, J. L. 
Wible, L. C. 


fai, RK. E. 
Bermetc, J. A. J. 
Biblarz, Q. 
Collins, DO. J. 
Gawain, T. H. 
Haupt, U. 
Ranr, C. H. 
Layton, D. M. 
Lindsey, G. H. 
Miller, J. A. 
Netzer, D. W. 
Platzer, M. F. 
Redlin, M. H. 
Senmiat, L. V. 
Vavra, M. H. 
mocker, R. D. 


Department of Business 


Administration and Economics 


1214 
Z29 
1236 
1247 
1258 
1269 
1270 
128] 
292 


1308 


Carrick, P. M. 
Church, W. H. 
Cowie, J. B. 
Darbyshire, L. 
Eisenhardt, P. 
Elster, Ri S% 
Fremgen, J. M. 
Gdilz,. J) G. 
Githens, W. H. 


Grainger, T. L. 


1319 Hoverland, H. A. 1636 Gardner, E. M. 1953 Gottschalk, S. 
1320 “Hynes, J. P- 1647 + Geist, J. NM. 1964 McAdams, J. i. 
saci y aces 1658 Gerba, A., dr. 1975 O'Neil, E. F. 
1342 Kushnick, S. A. 1669 Houston, R. K. 1986 Pearson, EW. 
1353 Lande, R. S. 1670 Kirk, D. E. 1997. Smith, B. M. L. 
1364 Lane, H. L. 1681, Klammae€. F., Jr. cOGG “Stolfigen eh: 
1375 Lee, M. 1692 Marmont, G. H. 2017 Terie Fate 

1386 ficdonaid, J. H. 1706 fhitier, R. L. beparwient or material 
1397 Moan, F. E. 1719 Murray, R. P. SClengerancsene maa 
1403 Noorzoy, S. 1720 Myers, G. A. 2028 Clark, J. R. 
1414 Smith, J. 1731 Myers, H. L. 2039 Helliwell, R. W. 
1425 Steckler, M. J. 1742 Oler, C. 8. oO ee SI eae 
1436 Vigen, J. W. 1753 Panholzer, R. 20S COs Se 
1447 Waibel, R. 1764 Rahe, G. A. SO eee 
1458 Wegener, W. H. 1775 Rothauge, C. H. 2073 Reinhardt, &. A. 
1469 sWiskoff, Mah. . I78om Sacknans G. c= Counce Uocmuuamie 
1470 Womer, N. K. 1797 Sheingold, A. sre re 
Department of Electrical 1803 Smith, W. C. 2101 Sinclair, J. &. 
Supncer ng aeecnsc ene: 2112 Tolles, W. M. 
1481 Adler, R. W. 1825 Stentz, D. A. Oa 
1492 Badger, R. occmes nD Department of Mathematics 
1508 Bauer, W. M. oc ener ee 2134 Bender, A. P. 
1519 Baycura, 0. M. 1858 Thaler, G. J. 2145 Bleick, W. E. 
1520 Borst, F. W., Jr. 1869 Titus, H. A. 2156 Bolles, R. C. 
1531 Bouldey, J. M. 1870 Turner, J. B., Jr. 2167 Budway, J. J. 
1542 Breida, S. 1881 Ward, J. R. 2178 Calabrese, P. G. 
1553) Campbell, J. D. 1892 Wilcox, M. L. 2189 Chewning, W. C. 
1564 3than, o- Department of Government 2190 Comstock, C. 
1575 Chaney, J. G. nea ene 2206 Davis, D. L. 
1586 Cooper, P. E. 1908 Alexander, W. P. 577 acne meee 
1597 Sto tton- MeL. 1919 Amos, J. W., II 2228 Estell, R. Jd. 
1603 De Laura, R. D. 1920 Bjarnason, L. LL. 2239 Faulkner, F. D. 
1614 Demetry, J. S. 1931: Boggess, W. C. 2240 Franke, R. 

1625 Ewing, G. D. 1942 Gabel, B. B. 2251 Giarratana, J. 
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Department 


Aeronautics 
Aviation Safety 
Bus Ad & Econ 
Elec Eng 

Govt & Humanities 
Mat Sc & Chem 
Math 

Mech Eng 
Meteorology 
Oceanography 

Ops Analysis 
Physics 

Navy Mange Sys Center 


List of Eligibie Faculty (all faculty engaged in teaching in the academic year (1970-71 


gcc 
22/3 
2284 
2295 
2301 
2312 
2323 
2334 
2345 
2356 
2367 
2378 
2389 
2390 
2406 
2417 
2428 
2439 
2440 
2451 
2462 
2473 
2484 
2495 
2501 
25lc 
2523 
2534 
2545 


Gibbons, G.. 0D. 
Hartkemeier, H. P. 
Hunt, R. W. 
Jayachandran, T. 
Jennings, W. 
Kiidalil 2G... A. 
Kodres, U. R. 
KOl1tz2; 6. t. 
Little, W. A. 

ETEz ler, le Ge 
Lucas, K. R. 
Marks, H. B. 
Morris, G. W. 
Bierce, J. P. 
Preisendorfer, R. W. 
Pulliam, F. M. 
Roberts, A. B. 
Schoenstadt, A. L. 
Schwarzkopf, A. B. 
Shorb, A. M. 
Sanger, E. A. 
Spalding, J. H. 
Stewart, E. J. 
Stoops, G. A. 
Syms, G. H. 
figanan, D. H. 
moe, ©. ESC. 
Weir, M. D. 

Wilde, C. 0. 


Department of Mechanical 
Engineering 


2556 
2507 
2578 
2589 
2590 
2606 
2617 
2628 
2639 
2640 
2651 
2662 


rock, J. E. 
Cantin, G. 
Rooper, |. E. 
Baurison, ©. J. 
Houlihan, T. M. 
Kelleher, M. D. 
mato, Ps. J: 
Newton, R. E. 
Nguyen, D. H. 
Prowell, R. W. 
Salinas, D. 
Winfrey, R. C. 


Department of Meteorology 


2073 
2695 
2684 
2701 
LINZ 
2723 
2734 
2745 
2756 
2767, 
2778 
2789 
2790 
3910 


Alberty, R. L. 
Carrigan, R. C. 
Davidson, K. L. 
Duthie, W. D. 

EV SDERCY y Rak. 
Hamilton, H. D. 
Mahiman, J. D. 
Martin, Ff. Ls 
Oakes, W. 
Senard t,0.02 ake 
TdVlOnyeeewes 
van der Bijl, W. 
Williams, R. T. 
Winninghoff, F. J. 


Department of Oceanography 


2806 
2817 
2828 
Ze o9 
2840 
2851 
2862 
2873 
2884 
2895 
2901 
Coie 
2923 
2934 
2945 
2956 
2967 


Andrews, R. S. 
BaSSe ttm Gv. 
BOStOn, IN. E- 
Crew, H. 
Denner, W. W. 
Galt, J. A. 
Geary, J. E. 
Giles, C. F. 
Hadertie, E. C. 
Jung aGaen. 

Smit thse weero!. 
VEeMmese tenn C. 
Thornton, E. 
Traganza, E. 
luckeres.- e- 
VORP OGnWINnd. su. cu. 
Wickham, J. B. 


Oo Ww 


Department of Operations 


Analysis 


2978 
2989 
2990 
3009 
3010 
302] 
3032 
3043 
3054 
3065 


Andrus, A. F. 
Arima, J. K. 
Barr, De R. 
Barrett, E. B. 
Bev Ean. 


Butterworth, R. W. 
Burnett, T. D. 
Calpiga id. Re 
Cunningham, W. P. 


Danskin, J. M., dr. 


3076 
3087 
3098 
3104 
Silo 
3126 
Sie 
3148 
3409 
3160 
371 
SoZ 
S193 
3209 
3210 
322] 
3232 
3243 
3254 
S200 
3276 
3287 
3298 
3304 
3319 
3326 
330/ 
3348 
S309 
3360 
337) 
soe 
3393 
3409 
3410 
342] 
3432 
3443 


ESdryare. 0. 
Ferguson, Rk. EE; 
FLOVdiew ooh: 
Free, W. D. 
Porrest, RaoiN. 
Gaver, D. P. 
Gieseke, W. J. 
Greenberg, H. 
Hartman, J. K. 
Hetdorns. G. c. 
Higgins, J. €. 
Howard, G. T. 
Jones, C. R. 
Kalman, P. Jd. 
Karamadian, S. 
Kochems, R. A. 
Larson. wl. 
Lindsey, G. F. 
McMasters, A. W. 
PareSiadd, Ke le 
MITChioes) Re 
Peterson, C. A. 
POOGK, Gk 
Preston, &. L. 
Richards, F. R. 
Schrady, D. A. 
Schwartz, tH. 
Shubert, B. Q. 
Shudde, kK. H. 
Sovereign, M. G. 
Tarter, M. J. 
Taytomend. G. 
Tuck, G. A. 
Tysver, J. B. 
Washburn, A. R. 
Woods, W. M. 
Zehna, P. W. 
Zweig, H. Jd. 


Department of Physics 


3454 
3465 
3476 
3487 
3498 
3504 


Armstead, R. L. 
BUSK likens aR. 
Ceglio, N. M. 
Cooper, A. W. 
Cooper, J. N. 
Coppens, A. B. 


Part D - Supporting remarks for your primary nomination (optional) 


S515 
3526 
Sia) ey7 
3548 
3559 
3560 
S571 
3582 
3395 
3609 
3610 
362] 
3632 
3643 
3654 


3665 
3676 
3687 
3698 
3704 
3715 


3726 


Crittenden, —.6.eae 
Danilizetheen. 
Ellen, Ace. 
Garrettson, G. A. 
Fain 1 ei wetleees 
Harrison, 0. (Esmee 
Kalmbach, S. H. 
Kelly, R. L. 
Kinsler, L. E. 
Epes oR aaGe 
Little, W. 
Medwin, H. 

Milne, E. A. 
Neighbours, J. R. 
Olsen, L. 0. 
Reese, W. 

Rigcin. J. D-. 
Rodeback, G. W. 
Sanders, J. V. 
Schacner, G. E. 
Schwirzke, F. R. 
Williamson, T. c. 
3737 Woehler, K. E. 


3/48 Zeleny, W. B. 


Navy Management Systen 
Center 


3752 
3760 
Sl 
3782 
5723 
3809 
3810 
382] 
3832 
3843 
3854 
3865 
3876 
3887 
3898 


3904 


Blandin, S. W. 
Boynton, R. E. 
Cantrerlaeces 
Childs. Fee. 
Dawson, J. E. 
De Serpa, A. C. 
Doce Ee 
rreed, ©). Uc 
Hartman, P. G. 
Keller, A. S. 
Mauer, W. A. 
Plotkin, N. 
Puscheck, H. C. 
von Pagenhardt, R. 
Ulrey, I. W. 


Wood, C. L. 
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